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Mass  Conservation  - 1 

Mass  conservation  is  important  for  climate  studies 

o  It  is  a  powerfull  debugging  tool  even  for  shorter 
time  scales 

Many  ocean  models  are  Bousinessq 

o  Density  differences  are  neglected  except  in  terms 
multiplied  by  g 

o  Implies  conservation  of  volume,  not  mass 
o  Still  want,  and  can  get,  tracer  conservation 

HYCOM  is  not  Bousinessq,  so  it  should  conserve 
mass 

o  Except  that  it  assumes  the  non-steric  SSH 
is  a  small  fraction  of  the  total  depth 

o  Includes  steric  effects,  such  as  mean  SSH 
rise  due  to  thermal  expansion,  but  does  not 
exactly  conserve  either  mass  or  volume 
o  Not  satisfactory  for  coastal  domains 
o  Replaces  dp  with  dp’  nearly  everywhere 


Mass  Conservation  -  II 

HYCOM’s  reputation  for  non-conservation  is  partially 
due  to  using  dp’  in  mean  calculations 

o  Much  better  conservation  properties  when 
correctly  using  dp  in  region-wide  means 

o  HYCOM  source  code  uses  dp’  for  means, 
i.e.  this  is  a  long  standing  “bug” 

New  option,  btrmas=1 ,  for  exact  mass  conservation 

o  From  Remy  Baraille  at  SHOM 
o  Removes  the  dp’  “equals”  dp  approximation 
o  Note  that  dp’  is  still  the  prognostic  variable 
o  Still  dp’  in  restart  and  archive  files 
o  Currently,  btrmas=1  is  less  stable  than  btrmas=0 
o  Still  working  on  making  it  more  stable 

As  a  test,  0.72°  Global  HYCOM  was  spun-up  for  5 
years  with  typical  atmospheric  forcing  and  then  all 
forcing  was  removed 

o  During  “spin  down”  there  should  be  no  change  to 
the  total  heat  and  salt,  i.e.  to  the  mean  T  and  S 
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Spin  Down  Test  of  Conservation  - 1 
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Spin  Down  Test  of  Conservation  -  II 
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Spin  Down  Test  of  Conservation  -  III 
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Robert- Asselin  Time  Filter 


One  potential  source  of  non-conservation  is  the  RA 
filter  used  to  stabilize  the  leapfrog  time  step  scheme 

o  Williams  (2009)  proposed  a  modified  filter  that  is 
more  conservative  and  more  accurate 

o  However,  it  is  not  applicable  to  models  with 
time  varying  layers  than  must  filter  h  and  hC 
consistently  while  maintaining  non-negative  fields 

Leclair  and  Madec  (2009)  showed  that  RA  is: 

o  Conservative  without  surface  forcing 

o  As  demonstrated  numerically  by  spin-down  case 

o  Can  be  made  conservative  if  surface  forcing  terms 
are  calculated  at  half  time  steps 

o  Implies  no  time  splitting  from  forcing 
o  Explicitly  remove  forcing  from  RA  filter 

HYCOM  is  not  currently  conservative  with  surface 
forcing 

o  Started  testing  Leclair’s  approach 


Bit-for-Bit  Multi-CPU  Reproducability 


Repeating  a  single  processor  run: 

o  Produces  identical  results 

Repeating  a  multi-processor  run: 

o  Produces  different  results 
o  Using  either  OpenMP  or  MPI 
o  e.g.  fastest  global  sum  is  non-reproducable 

o  Unless  programmer  explicitly  avoids 
non-reproducable  operations 

o  May  need  to  avoid  some  compiler  options 
Two  levels  of  reproducability 

o  On  the  same  number  of  processors 
o  Some  scalable  libraries  provide  this 
o  On  any  number  of  processors 
o  Only  “safe”  option  for  code  maintenance 

•  Always  requires  carefull  programming 

•  Can  be  slower 

o  Should  be  required  for  operational  ocean 
prediction  models 

•  Is  implemented  by  HYCOM 


Are  Two  HYCOM  Runs  Identical?  - 1 


The  only  way  to  comfirm  bit-for-bit  identity  is  to 
compare  binary  fields 

Could  compare  binary  archive  and/or  restart  files 

o  But  these  don’t  tell  you  where  any  differences 
came  from 

P-MICOM  used  “named  pipes”  to  compare  arrays 
between  MASTER  and  SLAVE  model  runs  while 
they  were  in  progress 

o  A  named  pipe  is  a  special  Unix  file  providing 
a  FIFO  capability  via  a  shared  memory  buffer 

o  Can  read  and  write  to  it  just  like  a  normal  file 

SLAVE  writes  an  array  to  the  pipe,  MASTER  reads 
the  array  and  compares  it  to  its  own  version 

o  Usually  MASTER  runs  on  one  processor  and 
SLAVE  on  multiple  processors 

o  Only  limitation  is  that  MASTER  and  SLAVE  must 
be  running  under  the  same  Unix  image 

o  May  be  difficult  to  arrange  for  MPI  on  a  cluster 


Are  Two  HYCOM  Runs  Identical?  -  II 


HYCOM  includes  a  named  pipe  based  comparitor 

o  Similar  to  P-MICOM,  but  easier  to  use 
o  Calls  to  compare  or  compareall  in  source  code: 

o  Can  trigger  a  comparison  of  arrays  at  run  time, 
between  two  HYCOMs  via  the  named  pipe 
o  Can  invoke  other  run-time  debugging  options 

A  new  option  is  to  compile  with  the  OCEANS2  macro 

o  Runs  two  instances  of  HYCOM  in  the  same 
executable 

o  Each  on  a  different  number  of  MPI  tasks 

o  Calls  to  compare  or  compareall  in  source  code: 

o  Will  trigger  a  comparison  of  arrays  at  run  time 
via  MPI  send/recv 

o  Easier  to  use  than  named  pipes  and  only 
requires  MPI 

o  Works  for  OpenMP  with  MPI,  but  same 
number  of  threads  used  by  both  HYCOMs 
o  Does  not  currently  work  in  coupled  models 


Tides  in  HYCOM  - 1 

Body  forcing  for  8  largest  components 

o  With  (optional)  nodal  corrections 

o  Implemented  in  HYCOM  by  NCEP 

Boundary  forcing  for  Flather  or  Browning-Kreiss  ports 

o  Implemented  by  various  groups  in  local  versions 
of  HYCOM 

o  Now  in  standard  version 

o  8  largest  components  specified  as  complex 
amplitudes  at  each  boundary  point  using 
unmodified  extracLHC  program  from  OSU’s 
OTPSnc  or  OTPS2  package 
o  Allows  for  curvilinear  grid 
o  With  (optional)  nodal  corrections 

o  Tidal  forcing  under  floating  ice  shelves  requires 
1 1 47  ports  for  Global  1  /1 2°  domain 

o  Port  implemention  updated  to  allocate 
memory  at  run  time  and  to  make  many  fewer 
MPI  calls  for  better  MPI  performance 


Tides  in  HYCOM  -  II 

Linear  tidal  drag  based  on  bottom  roughness 

o  Applied  to  near-bottom  tidal  velocity  or  to 
depth  averaged  tidal  velocity 

o  Tensor  drag  for  depth  averaged  case  only 

o  Use  a  lagged  49-hour  filter  as  the 
non-tidal  velocity 

o  Convolution  of  a  21  hour  Savitzky-Golay 
smoother  and  a  24.842  hour  boxcar  filter 
o  Passes  0.02%  of  semi-diurnal  and 
3.2%  of  diurnal  (1 .2%  of  total)  tides 

•  Replaces  a  lagged  25-hour  average 

•  Better  band  pass  and  better  diurnal  phase 
o  Limit  drag’s  e-folding  time  for  stability 

Self  Attraction  and  Loading 

o  “Scalar”  approximation: 
o  SAL  treated  as  a  fraction  of  non-steric  SSH 

•  Constant,  or  spacially  varying,  fraction 

o  Input  SAL  complex  amplitude  fields  from  a  file 
o  With  or  without  a  “scalar”  SAL 
o  Iterate  SAL  to  convergence 


Spacially  Varying  Self  Attraction  and  Loading 
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Self  Attraction  and  Loading  Comparison 
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Barotropic  Global  1/12°  M2-only  simulations 

o  Twin  cases  that  differ  only  in  Self  Attraction 
and  Loading 

The  percentage  of  the  globe  (Y)  where 
model  -  TPX08atlas  SSH  RMS  is  less  than  X  m 

o  Note  the  long  tail  with  the  median  (50%), 
for  the  with-SAL  cases,  between  3  cm  and  5  cm 

o  Median  is  typically  a  more  robust  statistic 
than  mean  or  global  RMS 


Tides  in  HYCOM  -  III 

Several  tide-specifc  diagnostic  programs: 

o  hycom_tidal_foreman 
o  Foreman  tidal  analysis  on  HYCOM  .a  file 
o  HYCOM’s  4096-word  blocking  allows 
strip-mined  transpose  from  (x,y,t)  to  (t,x,y) 

o  hycom_calcSAL 

o  Calculate  SAL  on  uniform  cylindrical  global  grid 
o  hycom_tidal_rms 

o  RMS  difference  between  two  sets  of  tides 
o  hycom_tidal_ap2ri  and  hycom_tidal_ri2ap 

o  Amp, Phase  to/from  Real, Imaginary  tidal 
components 

Tidal  analysis  enabling  output: 

o  HYCOM  SSH  has  mass  and  steric  anomalies 
o  Steric  SSH  can  optionally  be  output 
o  Steric  anomaly  plus  long  term  SSH  mean 
o  Explicitly  “filters”  external  tides 
o  Get  internal  tides  from  Foreman  tidal  analysis 
o  Non-steric  SSH  from  difference 
o  Largely  external  tides 


HYCOM  and  Sea  Ice 

Two-way  coupling  to  LANUs  CICE  sea  ice  model, 
regional  and  global  domains 

o  HYCOM  exports: 
o  SST,  SSS,  SSH 
o  Surface  Currents 
o  Available  Freeze/Melt  Heat  Flux 
o  CICE  exports: 
o  Ice  Concentration 
o  Ice-Ocean  Stress 

o  Actual  Freeze/Melt  Heat/Salt/Mass  Flux 
o  Solar  Radiation  at  Ice  Base 

o  Coupling  via  the  Earth  System  Modeling 
Framework 

o  ESMF  version  4.0.0rp2 
o  Plan  to  migrate  to  NUOPC  Layer  on  top  of 
ESMF  version  6.X.0r 

Coupled  to  CICE  version  4.0 

o  Version  4.1  was  released  in  May,  2010 

o  Plan  to  skip  4.1  and  implement  the 
“next  release”,  due  later  this  year 


Coming  Soon 


Wave  forcing 

o  Stokes  Drift  Current  (SDC) 
o  Wave-to-Ocean  Momentum  Flux  (WOMF) 
o  Bottom  Orbital  Wave  Current  (OWC) 

Wetting  and  Drying 

o  Made  possible  by  mass  conservation  option 

o  Code  from  Remy  Baraille  at  SHOM  is  already 
in  HYCOM  but  needs  more  testing 

Fully  region-independent 

o  Compile  once,  run  on  any  region  and  any 
number  of  processors 

o  Run-time  memory  allocation,  less  memory  used 
o  Reduces  performance 

•  Compilers  make  fewer  optimizations 

•  Prototype  is  5%  slower 

o  Needed  for  full  ESMF  compliance 

o  Single  executable,  multiple  components 
each  running  on  separate  opus 

o  HYCOM  arrays  currently  on  all  opus 


